A Study of Mobile Agent Platforms for Distributed Web Crawling

In the traditional centralized crawling techniques the pages from all over the web are brought to the search engine side which results a lot of unnecessary Internet traffic. In the distributed crawling with migrating agents, the mobile agent is send to the web server side that brings only required pages to the search engine side which reduces unnecessary overhead. The mobile (migrating) agents are processes capable of roaming a wide area networks, interacting with foreign hosts, gathering information on behalf of its owner and coming back having performed the duties set by its user. The mobile agents acquire the information from the local environment, filter locally and either stored with the agent or forwarded to some receiving destination. In this paper we focus on use of mobile agents in web crawling, various available options of mobile agent technologies and enhancements needed. KeywordsWeb search engine, migrating agents, agent platforms